Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(pipeline) Migrate from trusted.ci.jenkins.io to infra.ci.jenkins.io #37

Merged
merged 1 commit into from
Apr 25, 2022

Conversation

dduportal
Copy link
Contributor

@dduportal dduportal commented Apr 5, 2022

This PR migrates the build from trusted.ci into infra.ci as per jenkins-infra/helpdesk#2789

[edit (added after the 1s pass of review)]
A few notes:

  • The scripts requirements are expected to be met by the underlying agent template, which is https://github.com/jenkins-infra/docker-helmfile (ruby, ruby gems, shell script CLIs, jq, az CLI)
  • The "all in one" job was split in different multibranch, so each one can have its own lock and its own frequency and timeout
  • A default of "1 CPU + 256M" memory is defined per job. We'll see the impact in term of time for the permission-report. For the 4 others, it does not change anything compared to the beefy 8 vCPU / 16 GB machines on trusted.ci.

@dduportal
Copy link
Contributor Author

dduportal commented Apr 6, 2022

Hello @daniel-beck , I need your help on this one. I'm currently migrating "Artifactory Users" branch into its own pipeline, and the job fails when uploading the report into the bucket with the error ERROR: The specified blob already exists. (ref. https://infra.ci.jenkins.io/job/reports/job/artifactory-users-report/job/PR-37/30/execution/node/29/log/).

As per https://github.com/jenkins-infra/infra-reports#artifactory-users-report, I understand that the json report should be consumed when the RPU jobs runs (https://github.com/jenkins-infra/repository-permissions-updater/blob/master/src/main/groovy/io/jenkins/infra/repository_permissions_updater/KnownUsers.groovy).

But RPU seems to run every 3 hours (as per https://github.com/jenkins-infra/repository-permissions-updater/blob/master/Jenkinsfile#L17) while the actual infra-reports runs once per hour (https://github.com/jenkins-infra/infra-reports/blob/main/Jenkinsfile#L6) => I understand that the artifactory-report would be uploaded 2-3 times between 2 builds of RPU.
What did I miss exactly that prevent the upload to happen?

(edit) nevermind: the error is related to the az command line , which version changed. Just confirmed it by using the old version and it works.

@dduportal dduportal force-pushed the migrate-jenkins-to-infra.ci branch 4 times, most recently from 5180cef to 0345bf3 Compare April 7, 2022 16:56
@dduportal dduportal marked this pull request as ready for review April 7, 2022 18:01
@dduportal
Copy link
Contributor Author

OK, seems good to me. I need a careful review (what did I miss because for sure I missed something) + 24h to ensure that the reports are generated repeatly as expected.

Copy link
Member

@lemeurherve lemeurherve left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!
Some nitpicks, one blocking suggestion.
The indentation is all over the place, mixing spaces and tabs, will need some formatting.

artifactory-users-report/Jenkinsfile Show resolved Hide resolved
fork-report/Jenkinsfile Outdated Show resolved Hide resolved
jira-users-report/Jenkinsfile Show resolved Hide resolved
jira-users-report/Jenkinsfile Outdated Show resolved Hide resolved
artifactory-users-report/Jenkinsfile Outdated Show resolved Hide resolved
fork-report/Jenkinsfile Outdated Show resolved Hide resolved
maintainers-info-report/Jenkinsfile Outdated Show resolved Hide resolved
maintainers-info-report/Jenkinsfile Outdated Show resolved Hide resolved
permissions-report/Jenkinsfile Outdated Show resolved Hide resolved
maintainers-info-report/Jenkinsfile Show resolved Hide resolved
@lemeurherve lemeurherve self-requested a review April 7, 2022 19:21
@dduportal
Copy link
Contributor Author

@lemeurherve spaces fixed on Jenkinsfiles

@dduportal dduportal force-pushed the migrate-jenkins-to-infra.ci branch from 4e059b2 to 3353f5b Compare April 7, 2022 19:29
@lemeurherve
Copy link
Member

lemeurherve commented Apr 7, 2022

@dduportal WDYT about restoring the Dockerfile(s) so users can locally run these scripts without having to install their dependencies?

@dduportal
Copy link
Contributor Author

@dduportal WDYT about restoring the Dockerfile(s) so users can locally run these scripts without having to install their dependencies?

They should use the image jenkinsciinfra/helmfile instead. Is it ok if I add this to the readme?

@dduportal
Copy link
Contributor Author

@lemeurherve README updated + stage name updated so the github checks are easier to catch.

@dduportal
Copy link
Contributor Author

Closing the PR to avoid messing up with current trusted.ci. Will be reopen with fixes

@dduportal dduportal force-pushed the migrate-jenkins-to-infra.ci branch from 6115adb to ac9023f Compare April 11, 2022 16:23
@dduportal
Copy link
Contributor Author

Hello @zbynek @daniel-beck @NotMyFault @timja @lemeurherve

I've reopen the PR with the following notable changes (after the issues which caused jenkins-infra/plugin-site#1150):

  • Rolling back using jq version 1.5, as the alpine linux version is 1.6
  • Do not enable cron rebuild when not on the principal branch
  • Use a different report name when not on the principal branch (to allow comparing). Please note that there is no garbage colelcting on this one.

Copy link
Member

@NotMyFault NotMyFault left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't access infra. to verify the integrity of the generate files, but that sounds reasonable.

@dduportal
Copy link
Contributor Author

Update:

  • The report generated on the last version does not have the jq prepended which looks better. Reports looks valid syntaxically, but I reall need help to check it.
  • WDYT about disabling the publication step to the bucket when not on the principal branch? That would avoid the bucket being messy with a bunch of "*-PR-XX.json" files. Validation of the generated reports would be done through the archived artifact in PRs

@lemeurherve
Copy link
Member

WDYT about disabling the publication step to the bucket when not on the principal branch?

I'd say go for it.

@daniel-beck
Copy link
Contributor

Artifacts based comparison seems preferable.

@dduportal dduportal force-pushed the migrate-jenkins-to-infra.ci branch from 45931e2 to fedc626 Compare April 14, 2022 09:43
@dduportal dduportal changed the title Migrate jenkins to infra.ci chore(pipeline) Migrate from trusted.ci.jenkins.io to infra.ci.jenkins.io Apr 14, 2022
…s.io

Signed-off-by: Damien Duportal <damien.duportal@gmail.com>
@dduportal dduportal force-pushed the migrate-jenkins-to-infra.ci branch from 736abec to 1f3ab6c Compare April 25, 2022 07:25
@dduportal
Copy link
Contributor Author

Proceeding to merge + disabling former job in trusted.ci.
Gotta monitor the generated report and the RPUs runs all day long and tomorrow.

@dduportal dduportal merged commit cfd951c into jenkins-infra:main Apr 25, 2022
@dduportal dduportal deleted the migrate-jenkins-to-infra.ci branch April 25, 2022 07:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants